<<<<<<< HEAD

仅使用前视投影 得到的效果更好?可能是网络的规模不够大

尝试仅使用前视投影+sn


删除前视投影 仅保留pc分支:

loss非常不稳定 生成图像效果差


将pc分支的浅层特征经过down和res分支

另外将pc特征加入到discriminator中 或者从discriminator中取消投影输入


SN: spectral norm


在discriminator中删除conditional输入之后, feature matching loss和vgg loss的初始值变得非常高,这可能是因为没有conditional输入之后,discriminator在开始接受到的输入就完全是混乱的初始生成图像 没有与真实图像的feature比较类似的条件输入;但是loss下降的趋势还比较稳定(即没有造成训练过程的不稳定),后期也下降到和有conditional输入的discriminator一样的情况了


最终的测试结果没有加入conditional输入好


目前来看Generator输入的上采样图像的作用比较大


静止场景:

0016 0017 0019(步行街场景 行人多 光照复杂)


尝试noVGG 效果很差


尝试pixel shuffle 解决棋盘格效应


object sensitive loss


尝试对gamma进行随机增强,生成的图像亮度更高,在50轮的时候视觉效果更好一些,但是没有最开始的点云分支好


尝试加入pointnet2的head, 收敛速度变慢, 训练集有一些图像效果还可以,对训练集拟合效果不错


激光雷达+图像 去阴影


和相机重建出来的对比


将测试集换为tracking与object的差集 效果不错


尝试在resblock前加入pc特征 效果比resblock前加入的细节和边缘要好


对点云的intensity进行归一化


在Discriminator中取消SN后面的BN之后 Gan的Feature matching loss的量级下降了很大 从[7,12]下降到了[1, 2], 这可能是因为SN实现了Lipschitz连续条件 但是只使用SN时拟合效果非常差


无法复现之前pix2pix的效果了 目前的改动:Gamma:无效 flip:无效 batchsize 有一些效果,现在感觉可能是数据集的问题


尝试直接用前视投影的点云提特征?


加入膨胀卷积之后效果很好


远处的弱化 从loss上考虑



LiCAM segmentation 实验

Accumulating evaluation results...                                     │tmpfs      38G 3.8G  34G 10% /run

DONE (t=0.06s).                                              │/dev/sda2    439G 393G  24G 95% /

 Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.590              │tmpfs      189G 736M 188G  1% /dev/shm

 Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.882              │tmpfs      5.0M   0 5.0M  0% /run/lock

 Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.646              │tmpfs      189G   0 189G  0% /sys/fs/cgroup

 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.402              │/dev/sda1    511M 3.7M 508M  1% /boot/efi

 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.689              │tmpfs      38G  28K  38G  1% /run/user/108

 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.866              │tmpfs      38G  22G  16G 58% /run/user/1000

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.180              │tmpfs      38G   0  38G  0% /run/user/1001

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.613              │(base) ➜ sTrain-sTest_LiCAM_cas101_MS sudo kill 3771

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.625              │[sudo] password for bdbc201: 

 Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.453              │(base) ➜ sTrain-sTest_LiCAM_cas101_MS 

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.715              │(base) ➜ sTrain-sTest_LiCAM_cas101_MS du -sh

 Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.907    



LiCAM 测真实图像

DONE (t=0.20s).                                              │/dev/sda2    439G 393G  24G 95% /

 Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.613              │tmpfs      189G 736M 188G  1% /dev/shm

 Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.885              │tmpfs      5.0M   0 5.0M  0% /run/lock

 Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.689              │tmpfs      189G   0 189G  0% /sys/fs/cgroup

 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.397              │/dev/sda1    511M 3.7M 508M  1% /boot/efi

 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.737              │tmpfs      38G  28K  38G  1% /run/user/108

 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.886              │tmpfs      38G  22G  16G 58% /run/user/1000

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.195              │tmpfs      38G   0  38G  0% /run/user/1001

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.646              │(base) ➜ sTrain-sTest_LiCAM_cas101_MS sudo kill 3771

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.662              │[sudo] password for bdbc201: 

 Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.458              │(base) ➜ sTrain-sTest_LiCAM_cas101_MS 

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.778              │(base) ➜ sTrain-sTest_LiCAM_cas101_MS du -sh

 Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.932



真实图像测SPADE

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.449

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.678

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.485

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.224

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.582

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.783

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.161

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.481

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.481

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.254

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.610

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.816


真实图像测皮鞋pix2pixHD

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.146

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.301

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.130

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.038

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.179

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.407

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.092

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.168

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.168

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.043

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.209

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.442


真实图像测LiCAM

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.388 

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.636 

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.407 

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.151 

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.511 

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.809 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.152 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.429 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.431 

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.186 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.553 

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.847 



真实图像测SPADE-LICAM-Intensity

直接High res

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.371

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.586

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.410

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.111

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.502

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.813

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.141

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.402

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.402

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.137

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.532

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.858


Low res 2 High res

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.365

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.583

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.399

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.112

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.496

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.791

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.142

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.398

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.398

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.138

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.530

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.829


目前看应该是需要平滑的conditional input(这个地方可以abalation study)


真实图像测SAPDE-LICAM

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.337

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.550

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.362

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.102

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.453

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.767

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.135

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.376

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.376

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.128

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.494

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.814


真实图像测pix2pix

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.007 

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.010 

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.010 

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000 

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.004 

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.015 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.003 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.003 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.003 

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.003 

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.014 


真实图像测cycle

None 没有结果



改用OMP多核实现之后 生成1M个点可以从90s提高到10s


使用光栅化过程的zbuffer解决了深度遮挡的问题 可以考虑不用光线追踪 但是仍然是串行实现的


ray tracng 之后的结果会在属于同一个类别的大片联通区域产生孔洞,孔洞中的数值是与该区域的类别非常相似的数值。这在图像中无所谓,但是在semantic label里,相近id代表完全不同的类别,因此使用闭运算,先膨胀再腐蚀,可以很好的消除孔洞


闭运算后


闭运算前




**** SPADE 没有crop图像的上面120个pixel


U-Net或者hourglass 应用在point network中


点云segmentation 之后用结果对应图像的纹理做tracking



标定 想通过射线投影模型,把方差较大的点投影会拟合的平面上 但是由于x方向方差过大 拟合的标定板投影实际上出现了仿射变换 无法构成矩形棋盘


在分割棋盘格时,最开始是找Z轴方向密度变化最大的一个点作为分割点,但是由于horiozon扫出来的点扫镖器边缘密度不均匀,很容易在棋盘中间造成分割点。为了解决这个问题,每次找到Topk个可能的分割点,计算每个点与棋盘格长度的差,找到差距最小的那个点作为真正的分割点,这样可以保证这个点是可以覆盖整个棋盘的


尝试了用求得的外参反向计算内参,但是发现求得的结果还是和内参输入是一样的,这可能是因为固定了错误的内参,反向计算的内参都是在过拟合这个外参


对比手工标注和棋盘格标到相同的RMSE所需要的标定板摆放次数

** feature map在空间上进行泊松融合


轨迹叠加 追踪


* 寻找最优的摆放位置,可以先尝试暴力搜索,在给定的一组lidar-camera pair中找到一组子集,使得准确率最高



扰动LiDAR点云的坐标系,这样就构成了不同位置的点云-图像pair






legoloam 在狭窄走廊+行人遮挡会出现degeneration; 这个degeneration还和建图开始的位置有关, 因为不同的建图开始位置会导致不同的局部地图, 从而对mapoptimization的局部地图匹配造成影响;这个问题在lio-sam中更严重




BEV

BBA 在KITTI鸟瞰图上直接检测 效果特别差

AP@0.3只有0.14512388417054484

AP@0.5只有0.02



MotionNet


nuscene part1


train

1400短序列


val 170

Mean pixel classification accuracy: 0.9270618313934371

mean cat accuracy of Bg: 0.9455821795090201

mean cat accuracy of Vehicle: 0.5206497586391847

mean cat accuracy of Ped: 0.44901742677048573

mean cat accuracy of Bike: 0.0

mean cat accuracy of Others: 0.588819095477387

mean instance acc: 0.5008136920792154


test 500

Mean pixel classification accuracy: 0.9226152206242199

mean cat accuracy of Bg: 0.9450818408534926

mean cat accuracy of Vehicle: 0.574332482538324

mean cat accuracy of Ped: 0.5306154084638041

mean cat accuracy of Bike: 0.0

mean cat accuracy of Others: 0.5026899037205982

mean instance acc: 0.5105439271152438




livox horion的点云时间分布规律

每一帧的相对时间分布是有相位差的, 比如前25%收到的点, 并不在帧数据的前25%位置, 而是在整个点云中滑动. 这个滑动可能通过运动模型计算出来.


=======

仅使用前视投影 得到的效果更好?可能是网络的规模不够大

尝试仅使用前视投影+sn


删除前视投影 仅保留pc分支:

loss非常不稳定 生成图像效果差


将pc分支的浅层特征经过down和res分支

另外将pc特征加入到discriminator中 或者从discriminator中取消投影输入


SN: spectral norm


在discriminator中删除conditional输入之后, feature matching loss和vgg loss的初始值变得非常高,这可能是因为没有conditional输入之后,discriminator在开始接受到的输入就完全是混乱的初始生成图像 没有与真实图像的feature比较类似的条件输入;但是loss下降的趋势还比较稳定(即没有造成训练过程的不稳定),后期也下降到和有conditional输入的discriminator一样的情况了


最终的测试结果没有加入conditional输入好


目前来看Generator输入的上采样图像的作用比较大


静止场景:

0016 0017 0019(步行街场景 行人多 光照复杂)


尝试noVGG 效果很差


尝试pixel shuffle 解决棋盘格效应


object sensitive loss


尝试对gamma进行随机增强,生成的图像亮度更高,在50轮的时候视觉效果更好一些,但是没有最开始的点云分支好


尝试加入pointnet2的head, 收敛速度变慢, 训练集有一些图像效果还可以,对训练集拟合效果不错


激光雷达+图像 去阴影


和相机重建出来的对比


将测试集换为tracking与object的差集 效果不错


尝试在resblock前加入pc特征 效果比resblock前加入的细节和边缘要好


对点云的intensity进行归一化


在Discriminator中取消SN后面的BN之后 Gan的Feature matching loss的量级下降了很大 从[7,12]下降到了[1, 2], 这可能是因为SN实现了Lipschitz连续条件 但是只使用SN时拟合效果非常差


无法复现之前pix2pix的效果了 目前的改动:Gamma:无效 flip:无效 batchsize 有一些效果,现在感觉可能是数据集的问题


尝试直接用前视投影的点云提特征?


加入膨胀卷积之后效果很好


远处的弱化 从loss上考虑



LiCAM segmentation 实验

Accumulating evaluation results...                                     │tmpfs      38G 3.8G  34G 10% /run

DONE (t=0.06s).                                              │/dev/sda2    439G 393G  24G 95% /

 Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.590              │tmpfs      189G 736M 188G  1% /dev/shm

 Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.882              │tmpfs      5.0M   0 5.0M  0% /run/lock

 Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.646              │tmpfs      189G   0 189G  0% /sys/fs/cgroup

 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.402              │/dev/sda1    511M 3.7M 508M  1% /boot/efi

 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.689              │tmpfs      38G  28K  38G  1% /run/user/108

 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.866              │tmpfs      38G  22G  16G 58% /run/user/1000

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.180              │tmpfs      38G   0  38G  0% /run/user/1001

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.613              │(base) ➜ sTrain-sTest_LiCAM_cas101_MS sudo kill 3771

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.625              │[sudo] password for bdbc201: 

 Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.453              │(base) ➜ sTrain-sTest_LiCAM_cas101_MS 

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.715              │(base) ➜ sTrain-sTest_LiCAM_cas101_MS du -sh

 Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.907    



LiCAM 测真实图像

DONE (t=0.20s).                                              │/dev/sda2    439G 393G  24G 95% /

 Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.613              │tmpfs      189G 736M 188G  1% /dev/shm

 Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.885              │tmpfs      5.0M   0 5.0M  0% /run/lock

 Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.689              │tmpfs      189G   0 189G  0% /sys/fs/cgroup

 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.397              │/dev/sda1    511M 3.7M 508M  1% /boot/efi

 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.737              │tmpfs      38G  28K  38G  1% /run/user/108

 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.886              │tmpfs      38G  22G  16G 58% /run/user/1000

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.195              │tmpfs      38G   0  38G  0% /run/user/1001

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.646              │(base) ➜ sTrain-sTest_LiCAM_cas101_MS sudo kill 3771

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.662              │[sudo] password for bdbc201: 

 Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.458              │(base) ➜ sTrain-sTest_LiCAM_cas101_MS 

 Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.778              │(base) ➜ sTrain-sTest_LiCAM_cas101_MS du -sh

 Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.932



真实图像测SPADE

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.449

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.678

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.485

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.224

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.582

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.783

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.161

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.481

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.481

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.254

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.610

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.816


真实图像测皮鞋pix2pixHD

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.146

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.301

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.130

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.038

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.179

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.407

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.092

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.168

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.168

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.043

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.209

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.442


真实图像测LiCAM

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.388 

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.636 

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.407 

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.151 

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.511 

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.809 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.152 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.429 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.431 

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.186 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.553 

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.847 



真实图像测SPADE-LICAM-Intensity

直接High res

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.371

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.586

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.410

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.111

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.502

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.813

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.141

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.402

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.402

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.137

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.532

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.858


Low res 2 High res

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.365

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.583

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.399

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.112

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.496

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.791

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.142

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.398

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.398

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.138

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.530

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.829


目前看应该是需要平滑的conditional input(这个地方可以abalation study)


真实图像测SAPDE-LICAM

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.337

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.550

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.362

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.102

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.453

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.767

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.135

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.376

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.376

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.128

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.494

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.814


真实图像测pix2pix

Average Precision (AP) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.007 

Average Precision (AP) @[ IoU=0.50   | area=  all | maxDets=100 ] = 0.010 

Average Precision (AP) @[ IoU=0.75   | area=  all | maxDets=100 ] = 0.010 

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000 

Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.004 

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.015 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 1 ] = 0.003 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets= 10 ] = 0.003 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=  all | maxDets=100 ] = 0.003 

Average Recall   (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000 

Average Recall   (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.003 

Average Recall   (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.014 


真实图像测cycle

None 没有结果



改用OMP多核实现之后 生成1M个点可以从90s提高到10s


使用光栅化过程的zbuffer解决了深度遮挡的问题 可以考虑不用光线追踪 但是仍然是串行实现的


ray tracng 之后的结果会在属于同一个类别的大片联通区域产生孔洞,孔洞中的数值是与该区域的类别非常相似的数值。这在图像中无所谓,但是在semantic label里,相近id代表完全不同的类别,因此使用闭运算,先膨胀再腐蚀,可以很好的消除孔洞


闭运算后


闭运算前




**** SPADE 没有crop图像的上面120个pixel


U-Net或者hourglass 应用在point network中


点云segmentation 之后用结果对应图像的纹理做tracking



标定 想通过射线投影模型,把方差较大的点投影会拟合的平面上 但是由于x方向方差过大 拟合的标定板投影实际上出现了仿射变换 无法构成矩形棋盘


在分割棋盘格时,最开始是找Z轴方向密度变化最大的一个点作为分割点,但是由于horiozon扫出来的点扫镖器边缘密度不均匀,很容易在棋盘中间造成分割点。为了解决这个问题,每次找到Topk个可能的分割点,计算每个点与棋盘格长度的差,找到差距最小的那个点作为真正的分割点,这样可以保证这个点是可以覆盖整个棋盘的


尝试了用求得的外参反向计算内参,但是发现求得的结果还是和内参输入是一样的,这可能是因为固定了错误的内参,反向计算的内参都是在过拟合这个外参


对比手工标注和棋盘格标到相同的RMSE所需要的标定板摆放次数

** feature map在空间上进行泊松融合


轨迹叠加 追踪


* 寻找最优的摆放位置,可以先尝试暴力搜索,在给定的一组lidar-camera pair中找到一组子集,使得准确率最高



扰动LiDAR点云的坐标系,这样就构成了不同位置的点云-图像pair






legoloam 在狭窄走廊+行人遮挡会出现degeneration; 这个degeneration还和建图开始的位置有关, 因为不同的建图开始位置会导致不同的局部地图, 从而对mapoptimization的局部地图匹配造成影响;这个问题在lio-sam中更严重





BEV

BBA 在KITTI鸟瞰图上直接检测 效果特别差

AP@0.3只有0.14512388417054484

AP@0.5只有0.02




MotionNet


nuscene part1


train

1400短序列


val 170

Mean pixel classification accuracy: 0.9270618313934371

mean cat accuracy of Bg: 0.9455821795090201

mean cat accuracy of Vehicle: 0.5206497586391847

mean cat accuracy of Ped: 0.44901742677048573

mean cat accuracy of Bike: 0.0

mean cat accuracy of Others: 0.588819095477387

mean instance acc: 0.5008136920792154


test 500

Mean pixel classification accuracy: 0.9226152206242199

mean cat accuracy of Bg: 0.9450818408534926

mean cat accuracy of Vehicle: 0.574332482538324

mean cat accuracy of Ped: 0.5306154084638041

mean cat accuracy of Bike: 0.0

mean cat accuracy of Others: 0.5026899037205982

mean instance acc: 0.5105439271152438



vid2vid训练过程中,对flow分支的依赖比较高, flow收敛了之后generator loss会有大幅下降, 同时图像质量开始变高


对于horizon,投影到range image上是具有局部性的, 但是由如果忽略了叠加,会使得这个局部的range image变得稀疏


>>>>>>> 4694174fdc56d65a8d62ab822f885106d5c4e684